College students' mental health evaluation model based on tensor fusion network with multimodal data during the COVID-19 pandemic.

The COVID-19 pandemic has caused a series of effects on the mental health of college students, especially long-term home isolation or online learning, which has caused college students to have both academic pressure and employment pressure. How to accurately and effectively assess the mental health status of college students has become a research hotspot. Traditional methods based on questionnaires such as Self-Rating Depression Scale (SDS) and Self-Rating Anxiety Scale (SAS) are difficult to collect data and have poor evaluation accuracy. This paper analyzes the psychological state through text-images of multi-modal data with tensor fusion networks and constructs a mental health assessment model for college students. First, the validity of the model is verified through the MVSA (Multi-View Sentiment Analysis) dataset. Second, the psychological state of college students under the epidemic is analyzed using the collected text-images dataset. The results show that the TFN-MDA (Tensor Fusion Network-Multimodal Data Analysis) based mental health assessment model constructed in this paper can effectively assess the mental health status of college students, with an average accuracy of more than 70%.

A Multimodal Framework for the Identification of Vaccine Critical Memes on Twitter

Memes can be a useful way to spread information because they are funny, easy to share, and can spread quickly and reach further than other forms. With increased interest in COVID-19 vaccines, vaccination-related memes have grown in number and reach. Memes analysis can be difficult because they use sarcasm and often require contextual understanding. Previous research has shown promising results but could be improved by capturing global and local representations within memes to model contextual information. Further, the limited public availability of annotated vaccine critical memes datasets limit our ability to design computational methods to help design targeted interventions and boost vaccine uptake. To address these gaps, we present VaxMeme, which consists of 10,244 manually labelled memes. With VaxMeme, we propose a new multimodal framework designed to improve the memes' representation by learning the global and local representations of memes. The improved memes' representations are then fed to an attentive representation learning module to capture contextual information for classification using an optimised loss function. Experimental results show that our framework outperformed state-of-the-art methods with an F1-Score of 84.2%. We further analyse the transferability and generalisability of our framework and show that understanding both modalities is important to identify vaccine critical memes on Twitter. Finally, we discuss how understanding memes can be useful in designing shareable vaccination promotion, myth debunking memes and monitoring their uptake on social media platforms. © 2023 ACM.

Medical Small-scale Transportation Object Detection Algorithm Based on Multimodal Data Fusion

The emergence of the Covid-19 pandemic has greatly impact transportation, and unmanned transportation has been widely used in medical. The average precision of object detection as an important part in unmanned medical transportation. Object detection mainly relies on sensors of vehicles to obtain information about the surrounding obstacles like camera and LIDAR. In this paper, we introduce a new fusion way to fuse data from different modalities, as 2D and 3D object detection encouraging performance, they are typically based on a single modality and are unable to leverage information from other modalities. We leverage the geometric semantic consistency of 2D and 3D detection to obtain more accurate fusion results, and address the weaknesses of IoU in fusion network by using a generalized version as both a new loss and a new metric. The experimental evaluation on the challenging KITTI object detection benchmark, shows significant improvements in average precision, especially at bird's eye view metrics, which shows the feasibility and applicability of the network. © 2022 IEEE.

Data-Driven In-Crisis Community Identification for Disaster Response and Management

Since 2019, the world has been seriously impacted by the global pandemic, COVID-19, with millions of people adversely affected. This is coupled with a trend in which the intensity and frequency of natural disasters such as hurricanes, wildfires, and earthquakes have increased over the past decades. Larger and more diverse communities have been negatively influenced by these disasters and they might encounter crises socially and/or economically, further exacerbated when the natural disasters and pandemics co-occurred. However, conventional disaster response and management rely on human surveys and case studies to identify these in-crisis communities and their problems, which might not be effective and efficient due to the scale of the impacted population. In this paper, we propose to utilize the data-driven techniques and recent advances in artificial intelligence to automate the in-crisis community identification and improve its scalability and efficiency. Thus, immediate assistance to the in-crisis communities can be provided by society and timely disaster response and management can be achieved. A novel framework of the in-crisis community identification has been presented, which can be divided into three subtasks: (1) community detection, (2) in-crisis status detection, and (3) community demand and problem identification. Furthermore, the open issues and challenges toward automated in-crisis community identification are discussed to motivate future research and innovations in the area.

A Multimodal Data Analysis Approach to Social Media during Natural Disasters

During natural disasters, social media can provide real time or rapid disaster, perception information to help government managers carry out disaster response efforts efficiently. Therefore, it is of great significance to mine social media information accurately. In contrast to previous studies, this study proposes a multimodal data classification model for mining social media information. Using the model, the study employs Late Dirichlet Allocation (LDA) to identify subject information from multimodal data, then, the multimodal data is analyzed by bidirectional encoder representation from transformers (Bert) and visual geometry group 16 (Vgg-16). Text and image data are classified separately, resulting in real mining of topic information during disasters. This study uses Weibo data during the 2021 Henan heavy storm as the research object. Comparing the data with previous experiment results, this study proposes a model that can classify natural disaster topics more accurately. The accuracy of this study is 0.93. Compared with a topic-based event classification model KGE-MMSLDA, the accuracy of this study is improved by 12%. This study results in a real-time understanding of different themed natural disasters to help make informed decisions.

A Duo-generative Approach to Explainable Multimodal COVID-19 Misinformation Detection

This paper focuses on a critical problem of explainable multimodal COVID-19 misinformation detection where the goal is to accurately detect misleading information in multimodal COVID-19 news articles and provide the reason or evidence that can explain the detection results. Our work is motivated by the lack of judicious study of the association between different modalities (e.g., text and image) of the COVID-19 news content in current solutions. In this paper, we present a generative approach to detect multimodal COVID-19 misinformation by investigating the cross-modal association between the visual and textual content that is deeply embedded in the multimodal news content. Two critical challenges exist in developing our solution: 1) how to accurately assess the consistency between the visual and textual content of a multimodal COVID-19 news article? 2) How to effectively retrieve useful information from the unreliable user comments to explain the misinformation detection results? To address the above challenges, we develop a duo-generative explainable misinformation detection (DGExplain) framework that explicitly explores the cross-modal association between the news content in different modalities and effectively exploits user comments to detect and explain misinformation in multimodal COVID-19 news articles. We evaluate DGExplain on two real-world multimodal COVID-19 news datasets. Evaluation results demonstrate that DGExplain significantly outperforms state-of-the-art baselines in terms of the accuracy of multimodal COVID-19 misinformation detection and the explainability of detection explanations. © 2022 ACM.

Design, implementation and interpretation of algorithms to predict the progression of Alzheimer's disease

Alzheimer's Disease (AD) is a serious public health issue that results in significant social and financial burdens on the individuals and communities impacted. In order to tackle this public health crisis it is critical that the clinical and computational research communities collaborate to identify possible causes of this progressive memory disease. Close collaboration between these two communities has the potential to result in promising therapeutic treatments for AD and other health conditions. This dissertation presents a collection of algorithms and associated derivations designed to predict the progression of AD using multi-task and structured regularization techniques, clustering membership by way of nonnegative matrix factorization, and COVID-19 clinical outcome prediction using multi-instance learning methods. This work presents novel algorithms for handling multimodal and longitudinal data and details approaches for multitask and multi-instance learning techniques that can be applied in other fields. Extensive discussions on algorithm predictive performance, interpretability, and implementation are provided for each method and are designed to serve as a framework for future research. (PsycInfo Database Record (c) 2022 APA, all rights reserved)

MMCoVaR: Multimodal COVID-19 vaccine focused data repository for fake news detection and a baseline architecture for classification

The outbreak of COVID-19 has resulted in an "infodemic"that has encouraged the propagation of misinformation about COVID-19 and cure methods which, in turn, could negatively affect the adoption of recommended public health measures in the larger population. In this paper, we provide a new multimodal (consisting of images, text and temporal information) labeled dataset containing news articles and tweets on the COVID-19 vaccine. We collected 2,593 news articles from 80 publishers for one year between Feb 16th 2020 to May 8th 2021 and 24184 Twitter posts (collected between April 17th 2021 to May 8th 2021). We combine ratings from two news media ranking sites: Medias Bias Chart and Media Bias/Fact Check (MBFC) to classify the news dataset into two levels of credibility: reliable and unreliable. The combination of two filters allows for higher precision of labeling. We also propose a stance detection mechanism to annotate tweets into three levels of credibility: reliable, unreliable and inconclusive. We provide several statistics as well as other analytics like, publisher distribution, publication date distribution, topic analysis, etc. We also provide a novel architecture that classifies the news data into misinformation or truth to provide a baseline performance for this dataset. We find that the proposed architecture has an F-Score of 0.919 and accuracy of 0.882 for fake news detection. Furthermore, we provide benchmark performance for misinformation detection on tweet dataset. This new multimodal dataset can be used in research on COVID-19 vaccine, including misinformation detection, influence of fake COVID-19 vaccine information, etc. © 2021 ACM.

SafeCampus: Multimodal-Based Campus-Wide Pandemic Forecasting

The motivation of this work is to build a multimodal-based COVID-19 pandemic forecasting platform for a large-scale academic institution to minimize the impact of COVID-19 after resuming academic activities. The design of this multimodality work is steered by video, audio, and tweets. Before conducting COVID-19 prediction, we first trained diverse models, including traditional machine learning models (e.g., Naive Bayes, support vector machine, and TF-IDF) and deep learning models [e.g., long short-term memory (LSTM), MobileNetV2, and SSD], to extract meaningful information from video, audio, and tweets by 1) detecting and counting face masks, 2) detecting and counting cough for potential infected cases, and 3) conducting sentiment analysis based on COVID-19-related tweets. Finally, we fed the multimodal analysis results together with daily confirmed cases data and social distancing metrics into the LSTM model to predict the daily increase rate of confirmed cases for the next week. Important observations with supporting evidence are presented. © 1997-2012 IEEE.

Artificial intelligence in clinical care amidst COVID-19 pandemic: A systematic review.

The worldwide health crisis caused by the SARS-Cov-2 virus has resulted in>3 million deaths so far. Improving early screening, diagnosis and prognosis of the disease are critical steps in assisting healthcare professionals to save lives during this pandemic. Since WHO declared the COVID-19 outbreak as a pandemic, several studies have been conducted using Artificial Intelligence techniques to optimize these steps on clinical settings in terms of quality, accuracy and most importantly time. The objective of this study is to conduct a systematic literature review on published and preprint reports of Artificial Intelligence models developed and validated for screening, diagnosis and prognosis of the coronavirus disease 2019. We included 101 studies, published from January 1st, 2020 to December 30th, 2020, that developed AI prediction models which can be applied in the clinical setting. We identified in total 14 models for screening, 38 diagnostic models for detecting COVID-19 and 50 prognostic models for predicting ICU need, ventilator need, mortality risk, severity assessment or hospital length stay. Moreover, 43 studies were based on medical imaging and 58 studies on the use of clinical parameters, laboratory results or demographic features. Several heterogeneous predictors derived from multimodal data were identified. Analysis of these multimodal data, captured from various sources, in terms of prominence for each category of the included studies, was performed. Finally, Risk of Bias (RoB) analysis was also conducted to examine the applicability of the included studies in the clinical setting and assist healthcare providers, guideline developers, and policymakers.

An Emotion Care Model using Multimodal Textual Analysis on COVID-19.

At the dawn of the year 2020, the world was hit by a significant pandemic COVID-19, that traumatized the entire planet. The infectious spread grew in leaps and bounds and forced the policymakers and governments to move towards lockdown. The lockdown further compelled people to stay under house arrest, which further resulted in an outbreak of emotions on social media platforms. Perceiving people's emotional state during these times becomes critically and strategically important for the government and the policymakers. In this regard, a novel emotion care scheme has been proposed in this paper to analyze multimodal textual data contained in real-time tweets related to COVID-19. Moreover, this paper studies 8-scale emotions (Anger, Anticipation, Disgust, Fear, Joy, Sadness, Surprise, and Trust) over multiple categories such as nature, lockdown, health, education, market, and politics. This is the first of its kind linguistic analysis on multiple modes pertaining to the pandemic to the best of our understanding. Taking India as a case study, we inferred from this textual analysis that 'joy' has been lesser towards everything (~9-15%) but nature (~17%) due to the apparent fact of lessened pollution. The education system entailed more trust (~29%) due to teachers' fraternity's consistent efforts. The health sector witnessed sadness (~16%) and fear (~18%) as the dominant emotions among the masses as human lives were at stake. Additionally, the state-wise and emotion-wise depiction is also provided. An interactive internet application has also been developed for the same.

A multimodal deep learning-based drug repurposing approach for treatment of COVID-19.

Recently, various computational methods have been proposed to find new therapeutic applications of the existing drugs. The Multimodal Restricted Boltzmann Machine approach (MM-RBM), which has the capability to connect the information about the multiple modalities, can be applied to the problem of drug repurposing. The present study utilized MM-RBM to combine two types of data, including the chemical structures data of small molecules and differentially expressed genes as well as small molecules perturbations. In the proposed method, two separate RBMs were applied to find out the features and the specific probability distribution of each datum (modality). Besides, RBM was used to integrate the discovered features, resulting in the identification of the probability distribution of the combined data. The results demonstrated the significance of the clusters acquired by our model. These clusters were used to discover the medicines which were remarkably similar to the proposed medications to treat COVID-19. Moreover, the chemical structures of some small molecules as well as dysregulated genes' effect led us to suggest using these molecules to treat COVID-19. The results also showed that the proposed method might prove useful in detecting the highly promising remedies for COVID-19 with minimum side effects. All the source codes are accessible using https://github.com/LBBSoft/Multimodal-Drug-Repurposing.git.

AI-Driven Tools for Coronavirus Outbreak: Need of Active Learning and Cross-Population Train/Test Models on Multitudinal/Multimodal Data.

The novel coronavirus (COVID-19) outbreak, which was identified in late 2019, requires special attention because of its future epidemics and possible global threats. Beside clinical procedures and treatments, since Artificial Intelligence (AI) promises a new paradigm for healthcare, several different AI tools that are built upon Machine Learning (ML) algorithms are employed for analyzing data and decision-making processes. This means that AI-driven tools help identify COVID-19 outbreaks as well as forecast their nature of spread across the globe. However, unlike other healthcare issues, for COVID-19, to detect COVID-19, AI-driven tools are expected to have active learning-based cross-population train/test models that employs multitudinal and multimodal data, which is the primary purpose of the paper.